The disclosures of the foregoing applications are incorporated herein by reference in their entirety.
1. Field of the Invention
The present invention relates to networks, and more particularly, to reducing latency in a fibre channel arbitrated loop environment.
2. Background of the Invention
Fibre channel is a set of American National Standard Institute (ANSI) standards, which provides a serial transmission protocol for storage and network protocols such as HIPPI, SCSI, IP, ATM and others. Fibre channel provides an input/output interface to meet the requirements of both channel and network users.
Fibre channel supports three different topologies: point-to-point, arbitrated loop and fibre channel fabric. The point-to-point topology attaches two devices directly. The arbitrated loop topology attaches devices in a loop. The fibre channel fabric topology attaches host systems directly to a fabric, which are then connected to multiple devices. The fibre channel fabric topology allows several media types to be interconnected.
Fibre channel is a closed system that relies on multiple ports to exchange information on attributes and characteristics to determine if the ports can operate together. If the ports can work together, they define the criteria under which they communicate.
In fibre channel, a path is established between two nodes where the path's primary task is to transport data from one point to another at high speed with low latency, performing only simple error detection in hardware.
FC-AL is one fibre channel standard (incorporated herein by reference in its entirety) that establishes the protocols for an arbitrated loop topology.
In a Fibre Channel Arbitrated Loop (FC-AL) implementation of each device on the loop must have the intelligence to process incoming data in accordance with FC-AL rules. Latency occurs due to every device in the loop, as data is processed. This latency is accumulated around the loop. Even when a device is only required to pass data to the next device, it still has latency before passing on the data, because the device must continuously process data for changes in state. The accumulated latency slows down data transfers in a heavily populated loop.
The traditional FC-AL implementation has a transmitter of one device connected to the receiver of the next device and so on until the transmitter of the last device is connected to the receiver of the first device, creating a loop, as shown in FIG. 2.
In the traditional FC-Al implementation, all fibre channel devices (“FCDs”), to properly follow the FC-AL protocol, need to implement a LPSM. The LPSM has the following states: initializing, open-initiate, monitoring, arbitrating, arbitration won, open, opened, transmitted close, received close, transfer, and old-port (optional) before the FC-AL standard.
Under the FC-AL standard, there can be a maximum of 127 devices on a loop. Every device on the loop must be operating correctly for the loop to be operational. Because of this, many implementations include a bypass element that allows loop connectivity to be maintained when a particular device is not available for proper operation. Standard bypass elements are not intelligent devices and are not controlled based on data being transmitted but on device functionality or availability. These bypass elements do not add latency, but add jitter.
Each connected device must follow the FC-AL standard by decoding primitives, maintaining a Loop Port State Machine (LPSM), obtaining AL_PA's through the LIP process, performing arbitration, and sending frames in a manner that is consistent with the standard.
The following introduces some FC-AL process steps and how these processes add delay/latency:
LIP Process: In the traditional FC-AL implementation, the LIP process is started when one or more of the devices on the loop start sending LIPs to the next device in the loop. After all devices have seen the LIP, a temporary master is selected and this master sends out various frames that each device uses to select an AL_PA. The frames are sent in a serial fashion and the master must get the first frame back before sending the next type of frame. There are several trips around the loop before the LIP process is completed. Each device acquires an AL_PA and only needs to know its own AL_PA to be FC-AL compliant. Thus when all the frames return to the master, all the participating devices have their AL_PA's. The AL_PA's are used for identification and priority evaluation in the arbitration.
Each device in the loop takes time to process incoming data and deciding (per FC-AL standard) what to send out to its transmitter. Thus each device adds to latency and slows data processing. Two major points of latency are the time delay to create (and close) the loop circuit and the time delay for frames to be passed back and forth. To create a loop circuit in a traditional implementation the device must arbitrate for access and then open the device.
Arbitration process: In the traditional FC-AL implementation, the device desiring access to the loop does arbitration. In order to arbitrate and win arbitration, the device must send arbitration primitives and receive the arbitration primitive back.
In the simplest case, only one device wants access. The device that wants access to the loop sends its arbitration primitive to the next device in the loop and after some processing the next device passes the arbitration primitive onto the next device and so on until the primitive returns to the originating device. Therefore, for the FCD to know it has won arbitration there is an entire loop delay, even if it is the only device that wants access to the loop.
Open process: In the traditional FC-AL implementation to complete a loop circuit, the winning device needs to open the device it wants to communicate with. To open the other device and know that it is open (using zero BB_Credit) the device sends an open primitive and receives back an R_RDY primitive. The open primitive (OPN) is sent by the winning device to the next device, which processes it and sends it on to the next device and so on until the destination device receives the open primitive. Then the destination FCD sends a R_RDY primitive to the next device it is connected to and that device processes the primitive and sends it to the next device and so on, until the R_RDY primitive reaches the winning FCD. Thus to open the loop circuit and to know it is open, causes a delay around the loop.
“Open Replicate” process: In the traditional FC-AL implementation, to complete the loop circuit a winning FCD can send an open replicate primitive(s). When the winning FCD does a broadcast open replicate then every FCD will replicate the incoming frames and send them out to the next device. When the winning FCD does a specific open replicate then it sends out a number of open replicate primitives to inform the FCDs (that it wants to communicate) to replicate the frames on the incoming port and also send the frame on to the next device. There are no R_RDY primitives sent for an open replicate primitive, so frames can quickly follow the open primitive(s), but there is still delay to pass the frames around the entire loop.
Close process: In the traditional implementation to close the loop circuit, active devices must send a close primitive to the other device. An entire loop delay occurs when one of the devices sends a close primitive that is received by the other device before it sends its close primitive. Once the first device receives the second close primitive then the loop circuit is closed. Per the FC-AL standard, it is not required for the close primitives to be sent in a serial fashion, hence, the close primitives can be traveling the loop at the same time affecting the overall latency.
The following summarizes, the problems with traditional FC-AL implementation:
Problem 1: Large latency in loop communication implementations.
Problem 2: Separate devices to bypass non-working devices add cost to overall system and add jitter to the communication network.
Problem 3: Since only a device needs to know its AL_PA in a conventional standard system, there is no required “system” for determining the AL_PAs used in a loop.
Problem 4: Conventional methodology does not support global arbitration.
Problem 5: All devices on a loop currently need to support a full LPSM, and hence can be expensive.
Problem 6: The open replicate feature of the FC-AL has a long latency on frames sent because all frames need to return to the source device to be removed from the loop.
Therefore, what is required is a process and system that can accelerate frame processing and minimize latency in fibre channel arbitrated loop topology.